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SUMMARY 


A quasi-experimental method and test-retest design were used to 
explore the impact of participating in test preparation, whether the 
impact of test preparation depends on the first ACT® score, and the 
impact of specific test preparation activities. Test preparation 
improved students’ retest scores, and this effect did not differ 
depending on students’ first ACT score. Among specific test prep 
activities, only the number of hours using a private tutor resulted in 
increased score gains above the overall effect of test prep. 
Students who reported feeling inadequately prepared for the 
second test had ACT Composite scores that were lower than those 
students who felt adequately prepared. 


SO WHAT? 


Performance on high-stakes, standardized college entrance exams 
(e.g., SAT, ACT) provides important information that is used by 
postsecondary institutions in their admission process. Higher exam 
scores increase the chances of a student getting admitted in more 
selective schools and eligibility for merit-based scholarships. This 
said, students and their families are spending time and resources to 
prepare for these assessments. It is important to provide research- 
based evidence as to which types of test preparation activities work. 


NOW WHAT? 


Given the financial burden test preparation can place on families, it is 
imperative to understand what kind of impact is reasonable to expect. 
We would like to continue this work by investigating what types of 
activities students engage in, and among those activities, which are 
most efficacious. 
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College Entrance Exams: How Does Test 
Preparation Affect Retest Scores? 


Raeal Moore, PhD, Edgar Sanchez, PhD, and Sweet San Pedro, PhD 


Introduction 


Performance on high-stakes, standardized college 
entrance exams (e.g., SAT, ACT®) provides 
important information that is used by 
postsecondary institutions in their admission 
process (Clinedinst, 2014). Higher exam scores 
increase the chances of a student getting 
admitted in more selective schools and eligibility 
for merit-based scholarships (Carnevale & Rose, 
2003; Doyle, 2006). Given this context, test 
preparation programs have emerged to address 
students’ desire to improve their test scores 
whether by “improving the skills measured by the 
test or by improving the skills for taking the test, 
or both” (Messick, 1982). Test developers and 
other service providers have developed a number 
of test preparation solutions, including 
workbooks, practice tests, coaching, class 
curricula, as well as in-person or online tutoring 
packages. These solutions can be characterized 
by instruction aimed at developing skills and 
abilities, practice and exposure to practice 
problems that resemble the actual text, as well as 
the awareness and practice of test-taking 
strategies. Additionally, test preparation activities 
vary by duration and intensity. 


Despite the growth of the test preparation industry 
(Barnes Reports, 2017), the existing research on 
the efficacy of test preparation has shown mixed 
results depending on the type of test preparation 
activity, population tested, or study 
design/methodology (Lane, Raymond, & 
Haladyna, 2015). On average, research shows 
test preparation has small to moderate positive 
effects on actual test scores, with average test 
score gains falling within the margin of error for 


standardized college entrance exams. A study by 
Briggs (2001) showed test preparation solutions 
such as commercial preparation classes and 
tutoring had very small effects on ACT subject test 
scores, in which score increases did not exceed a 
full score point. The National Association for 
College Admissions Counseling (NACAC) 
reviewed efficacy studies on test preparation and 
also found minimal effects on test scores—an 
average gain of approximately 30 points on the 
SAT (Briggs, 2009). Meta-analyses on test 
preparation for the SAT or similar achievement 
tests show a modest test score gain of 0.25 
standard deviations (Bangert-Drowns, Kulik, & 
Kulik, 1983). Assuming similar gains for the ACT 
test, this equates to a gain of about 1.3 to 1.7 
points for the subject tests and the Composite 
score (Montgomery & Lily, 2012; Powers, 1993). 


Irrespective of engaging in test preparation 
activities, research has found that students who 
retest tend to increase their score (Appelrouth, 
Zabrucky, & Moore, 2015). For example, a 
meta-analysis found increases in test scores of 
approximately 0.25 standard deviations for 
students who retook a cognitive ability test when 
assessed between the first and second 
administration (Hausknecht et al., 2007). For the 
ACT test, retesting has shown a 1 to 2 point 
scale score increase, on average (Andrews & 
Ziomek, 1998). The exact gain associated with 
retesting is dependent upon a number of factors, 
including the number of times a student has 
tested, in what grade a student takes the ACT 
test, and how much time has elapsed between 
test administrations (ACT, 2017). 
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Aside from efficacy studies, research on test 
preparation has also looked into practice 
opportunities and motivational factors that may 
result in test score gains. Appelrouth, Zabrucky, 
and Moore (2015) used regression analyses to 
examine the relationship between SAT score gains 
and several factors of test preparation sessions. 
Their findings showed that time spent on individual 
tutoring was positively associated with an increase 
in the SAT Total score, with each additional hour 
spent on individual tutoring increasing the final SAT 
score by 2.34 points.’ In addition, the number of 
months students use test preparation was also 
positively associated with SAT score gains: 
Students who start test preparation before June of 
their junior year would stand to gain 4.3 points for 
every proceeding month. As such, a student who 
would start preparing for the SAT in February of 
their junior year would score 34.4 points higher 
than a student who would start preparing in 
September of their senior year (Appelrouth, et al., 
2015). 


For the ACT test, research also shows that test 
preparation is only associated with modest gains in 
ACT test scores, with higher increases related to 
length of preparation. Students who took and 
prepared for a second ACT test gained an average 
of 1.4 points on their second ACT Composite score 
(Schiel & Valiga, 2014a). Students who prepared 
for the second test for over 20 hours attained 
Composite scores 0.7 point higher, on average, 
compared to those who reported three to six hours 
of preparation (Schiel & Valiga, 2014b). Short-term 
preparation activities (e.g. commercial workbooks) 
were found to be associated with 1.2 to 1.5 point 
increases in ACT Composite score, while longer- 
term learning activities (e.g. high school 
coursework) were associated with 2.5 to 5.8 point 
increases in ACT Composite score (ACT, 2005). 


In addition to factors related to practice 
opportunities (i.e., length of test preparation), 
studies have also looked into the relationship 
between motivational factors and test performance 
in students’ test preparation and test-taking, such 
as awareness of their test preparedness or their 
self-efficacy. Mulvenon, Stegman, and Ritter (2005) 
evaluated the impact of anxiety, pressure, and self- 
efficacy on student performance on standardized 


tests. They found students’ perceptions of negative 
pressure about testing negatively influenced their 
test performance, while students’ perceptions of 
their ability in math and reading were positively 
associated with their performance in math and 
reading assessments. Hong, Sas, and Sas (2006) 
looked into awareness of students’ own level of 
preparedness and whether there were differences 
between high- and low-achieving high school 
students for an algebra test. Their findings show 
that high-achieving students were more aware of 
their test preparedness in cognitive areas 
(competence, understanding of the material, and 
study behavior). Peng, Hong, and Mason (2014) 
modeled the relationships among motivational 
variables in test-taking and test preparation 
strategies of 10th graders and found students with 
high self-efficacy applied more effort in test-taking 
and performed better than students with low self- 
efficacy. Findings in these studies are consistent 
with self-efficacy having a positive influence on 
student achievement (Bandura, 1993). 


Although numerous and extensive, existing studies 
that have looked at effects of test preparation on 
SAT or ACT test performance have some 
limitations. According to the previously mentioned 
NACAC report (Briggs, 2009), more than 30 
studies have been conducted since 1953 
evaluating the effect of coaching on the SAT. Only 
two such studies had examined the effect of 
coaching on the ACT, and few were conducted on 
students taking the tests since 2000. Many of the 
previous studies also used small samples that 
were not necessarily representative of high school 
students taking the SAT or ACT (Briggs, 2009) or 
used research designs that raised concerns about 
either internal or external validity (Lane et al., 
2015). Likewise, recent work (i.e., Appelrouth et 
al., 2015) focused on students who paid for 
services at select SAT test preparation centers. As 
test preparation becomes widely accessible 
through different delivery systems, large-scale 
studies of test preparation efficacy that involve a 
variety of test preparation activities become more 
important to understanding the value and impact of 
test preparation activities on both the ACT and 
SAT. 
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In this paper, we examine the impact of 
participating in test preparation prior to retaking 
the ACT test. The present study aims to address 
the call by Briggs (2009) for additional and more 
robust research on the efficacy of test 
preparation programs for the ACT. This study 
used a fairly large sample of ACT-tested 
students. For the first research question, we 
used a quasi-experimental deign that takes 
advantage of propensity score matching 
techniques to make causal claims related to test 
preparation effectiveness. The second research 
question used traditional regression techniques 
with covariates used to account for group 
differences. 


@ Using a pretest-posttest design, do 
students who participate in test 
preparation have larger score gains 
relative to students who did not 
participate in test preparation; does the 
test preparation effect depend on 
students’ pretest scores? 


& Among students who participated in test 
preparation, is the number of hours spent 
participating in each of 10 test preparation 
activities related to retest scores? 


3.) Among students who participated in test 
preparation, do their own beliefs that 
they might have been ill-prepared to 
take the test, regardless of the test 
preparation activities they engaged in, 
impact retest scores? 


Unlike prior observational studies about test 
preparation for the ACT, the present study 
utilizes a quasi-experimental method, applying a 
more rigorous design that will allow us to make 
causal inferences about efficacy of test 
preparation and significant factors that may 


result in higher test scores. Issues of the fairness 
of ACT score use in college admissions will also 
be explored in light of our findings. We focus on 
the effects of test preparation on ACT Composite 
score as it is most commonly used and 
referenced by students and admissions offices 
alike. 


Method 
Data 


The data for this study were taken from a 
previously conducted study on test preparation 
(Schiel & Valiga, 2014a; Schiel & Valiga, 2014b). 
In that study, an online survey was administered 
to a random sample of 76,000 ACT test-takers 
who had completed the ACT for the first time in 
April or June 2012 and retested in October 2012 
or completed the ACT for the first time in 
September, October, or December 2012 and 
retested in April 2013. These students were 
invited via email to participate in a survey about 
their test preparation activities. Of these 
students, 9,654 students responded to the 
survey (12.7% response rate). 


We were interested in students that had retested 
for the ACT, had not used test preparation aids 
prior to their first test, but had used test 
preparation aids prior to their second test. These 
selection criteria and the use of propensity score 
matching (PSM; detailed in the Data Analysis 
section) resulted in a total number of 2,660 
students in the analysis sample. Of these, 1,330 
students indicated that they did prepare for their 
second test, and another 1,330 students 
indicated that they did not prepare for their 
second test and thus were selected as a 
comparison group. Student self-reported 
demographic and background information (e.g., 
race, parental income) as well as academic 
performance indicators (e.g., high school GPA) 
were provided by students at the time of ACT 
test registration. 
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Participants 


Of the 2,660 students used in the analysis for 
this study, most were female (62%), in their 
junior year of high school (85%), and White 
(66%) (See Table 4). Approximately 47% of 
study participants were middle income 
($36,000—$100,000) with fewer percentages of 
students coming from lower- and higher-income 
families. The study sample closely resembled 
the population of 2013 ACT test-takers (58% 
White, 13% African-American, 15% 
Hispanic/Latino, 4% Asian, and 4% other/multi- 
race) but differed somewhat on gender (58% 
female, 42% male; ACT, 2015). Survey 
respondents also had a slightly higher ACT 
Composite test score (M = 22.5, SD = 4.7) than 
the national average (M = 20.9, SD = 4.8). 


Measures 


Survey of Test Preparation Activities 
for the ACT 


The ACT test preparation survey consisted of 
three sections that measured test preparation, 
how well students thought they were prepared, 
and amount of time allocated to 10 preparation 
activities. From this survey, data on test 
preparation activity, lack of academic preparation 
for the ACT, and amount of test preparation the 
student participated in were collected. 


Test Preparation 


Students were asked to indicate whether or not 
they prepared for the first and/or second ACT 


test administration. This survey question was 
written such that we asked students about test 
preparation activities “outside of normal 
classroom participation.” 


Lack of Preparation 


This measure consisted of students’ answers to 
three survey items: “I have not yet taken the 
class(es) necessary for doing well on one or 
more areas of the test”; “Some areas of the test 
had not been covered at all or had not been 
covered adequately in my high school classes’; 
and “I realized | had not done anything to 
prepare myself for taking this type of test.” For 
each item, students were instructed to answer 
yes or no. If a student said yes to any one of the 
three questions, they were coded as perceiving 
themselves to have a lack of preparation for the 
ACT test. 


Amount of Test Preparation 


The survey also asked students about their 
exposure to 10 specific test preparation 
activities. These activities ranged from 
interaction with online test materials to self- 
directed workbooks to one-on-one test 
preparation (Table 1). Students were asked to 
indicate the number of hours they spent on each 
of the activities in preparation for the second 
test, using a four-point scale (0 = 0 hours spent 
on activity; 1= 1-5 hours; 2 = 6-10 hours; 3 = 11 
or more hours spent on activity). Example items 
include: “Worked with a private tutor or 
consultant” and “Took a commercial test- 
preparation course(s).” 


Table 1. Test Preparation Activities Investigated in the Survey 


Test Preparation Activities 


Practice Test in ACT’s Free Preparing for the ACT 
ACT Online Prep 

Another Web-Based Test-Preparation Program 
The Real ACT Prep Guide 

Another Test Prep Workbook 


Test Prep Workshops/Courses Offered by the High School 


Commercial Test Prep Course 

Private Tutor or Consultant 

One-on-One with a High School Teacher 
Other Test Prep Software 
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ACT Composite Score 


The ACT is a curriculum-based battery of four 
multiple-choice tests of educational 
achievement—English, mathematics, reading, 
and science—and an optional writing test. The 
ACT, typically taken in the 11th or 12th grade, 
measures students’ academic readiness for 
college in key content areas. The ACT 
Composite score is the arithmetic mean of the 
four subject test scores rounded to the nearest 
whole number and reported on a scaled score 
from 1 to 36. 


Statistical Controls 


In order to better isolate the effect of test 
preparation, we included additional student 
characteristics in our analysis. Each student 
characteristic was included because of the 
theoretical relationship it has with ACT 
performance (Appenrouth et al., 2015; Radunzel 
& Noble, 2012; Sanchez, 2013). In this study, 
we statistically controlled for the impact of 
whether students commonly experience test 
anxiety; prior ACT Composite score; whether the 
student took at least trigonometry in high school; 
whether the student took at least physics in high 
school; whether advanced placement courses in 
mathematics, science, and English were taken 
in high school; high school GPA; whether 
students felt they needed assistance in core 
content area(s); whether high school attended 
was public or private; whether a fee waiver was 
used to take the ACT; highest parental 
education level; student’s grade level; student’s 
gender and race; and parents’ income. Most of 
these student academic and demographic 
characteristics were collected at the time of 
registration while information about test anxiety 
was collected from the survey. 


Data Analysis 
Propensity Score Matching (PSM) 


In studying the effect of test preparation, the 
interest is in estimating the causal effects of 
engaging in test preparation while controlling for 
confounding variables, such as parents’ income 


and prior student achievement. However, random 
assignment in educational settings raises ethical 
and logistical concerns (Walker, Hoggart, & 
Hamilton, 2008). Propensity score matching 
presents an alternative to randomization that 
models the assignment of students to the 
treatment conditions: in this case, the assignment 
of students to either receive test preparation or 
not, using an estimated propensity score 
(Rosenbaum & Rubin, 1983; see Nagengast, 
Marsh, & Hau, 2013, for a discussion of how 
propensity score matching is a better alternative 
to covariate adjustment). This estimated 
propensity score measures the probability of 
being assigned to the test preparation group 
given the covariates entered into the propensity 
score matching analysis. Through this process, 
we created control and treatment groups that are 
comparable on the propensity score which 
incorporates all included covariates. 


In this study, we used the SAS software macro 
“OneToManyMTCH?” (Parsons, 2004). This macro 
uses a greedy, nearest-neighbor matching 
algorithm to identify a matched sample for the 
control group, using an eight to one-digit match. 
We used one-to-one matching between treatment 
and control groups. The calculation of the 
propensity score was based on a logistic 
regression model that included 13 related 
covariates. Of those, 10 covariates were retained 
using stepwise selection and forced inclusion 
(Table 2).? 


Linear Regression 


Two linear regression models were estimated to 
address the research questions. The first 
regression analysis sought to determine if 
students who participate in test preparation 
activities have a higher ACT Composite retest 
score than those who did not prepare for the test, 
controlling for academic and demographic 
characteristics. These academic and 
demographic characteristics were included in the 
propensity score model and balanced by 
matching. We include them here in an effort to 
better capture the relationship between 
potentially confounding variables, activities of 
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interest, and our outcome of ACT test score. 
Interactions between preparation status and 
demographic information, and between the 
students’ first ACT score and test preparation 
status, were tested. This allowed us to test 
whether there is a differential impact of 
preparation for student subgroups (i.e., race, 
gender, and parent income) and across the range 
of prior ACT Composite scores. 


The second regression model, including only 
those students who participated in test 
preparation, sought to determine whether the test 
preparation effect identified in the first research 
question varied by the number of hours of 
participation. In addition to the variables entered 
into the model for the first research question, we 
included in this second regression model the 
number of hours spent on 10 test preparation 
activities and an indicator for whether students 
thought they were ill prepared to take the second 
test, regardless of the preparation activities 
engaged in. The 10 test preparation activities 
were treated as continuous measures in the 
regression model since dummy coding each 
activity for four categories would have resulted in 
estimating many coefficients relative to the 
reduced sample size used for research question 
two. For those test preparation activities found to 
be statistically significant in the linear model, pair- 
wise comparisons across the number of hours 
participating in the test preparation activity were 
conducted. 


| 


Results 
Propensity Score Matching 


Table 3 illustrates the balancing that was 
achieved in the control and treatment groups by 
using propensity score matching. This table 
shows the standardized mean difference between 
groups on each of the covariates investigated. 
Prior to matching, large group differences were 
observed for minority group membership, 
parental income, highest parental education, test 
anxiety, using a fee waiver, and high school GPA. 
After matching, the standardized mean 
differences between the treatment and control 
groups for all covariates were well below 10%. 


For continuous variables, the standardized mean 
X1-Xp 
2453’ 
2 
X, and X, denote the treatment and control 
sample means, and s? and s3 denote the 
corresponding sample variances. For binary 
variables, the standardized mean difference is 


difference can be estimated as d = where 


(P1—P2) 
[exGrPalsPate-Pa)l 
proportions of the treatment and control groups. 
An extension of this binary case is available, 
which makes use of multivariate Mahalanobis 
distance to handle multinomial variables (Yang & 
Dalton, 2012). 


d= where P, and P; denote the 


In addition to this empirical estimation of 
balance, the distributions of each covariate were 
examined. For illustrative purposes, the 
distribution of the propensity score for the 
treatment and control groups after matching are 
displayed in Figure 1. The distributions for both 
groups are very similar. Similar distributions 
were found when examining other continuous 
variables and similar proportions were found 
across groups for categorical variables. 
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Table 2. Student Characteristics Considered and Retained in the Propensity Score Model 


Retained in 


Propensity 


SYoxo) g= WY Foye (=) | 
Y 


& S&S NON NS NS NON 


Student Characteristic 

Prior ACT Composite score 

High school GPA 

Interaction between prior ACT Composite and high school GPA 
Gender (Female) 

Minority membership (African American, Hispanic, American Indian, or Native Hawaiian) 
Family Income 

Highest parental Education 

Self-reported experience of anxiety during testing 

Fee waiver status indicator 

Taken math coursework beyond Algebra II 

Taken Science coursework beyond Chemistry 

Taken advanced mathematics coursework 

Taken advanced science coursework 

Taken advanced English coursework 


Note: To qualify for an ACT fee waiver, a student must be currently enrolled in the 11th or 12th grade; a US citizen or testing in the 
US, US territories, or Puerto Rico; and must meet one or more indicators of economic need listed on the ACT Fee Waiver form. 
These students are eligible to receive a fee waiver to cover the cost of taking the ACT. 
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Table 3. Propensity Score Matching Balancing Results 


Pre-Match Pre-Match Pre-Match Post-Match Post-Match Post-Match 
Treatment Foy alice) | Pre-Match Standardized Treatment OFey al ine) | Post-Match Si clalel-lee|ra-ve| 
Mean Mean Difference Difference (%) Mean Mean Difference Difference (%) 
Minority Status 0.17 0.26 -0.10 -23.49 0.21 0.22 -0.01 -1.84 
Parental Income 2:18 1.94 0.24 32.18 2.08 2.05 0.03 3.85 
Male 0.65 0.62 0.03 6.73 0.64 0.60 0.03 6.97 
Highest Parental Education 5.59 4.96 0.63 31.72 5.29 5.25 0.03 1.74 
Test Anxiety 0.30 0.24 0.06 13.53 0.27 0.27 0.00 0.85 
Fee Waiver 0.16 0.29 -0.13 -31.71 0.20 0.22 -0.02 -4.08 
Taken Algebra 2 or Beyond 0.56 0.58 -0.02 -4.13 0.55 0.56 -0.01 -2.57 
Taken Physics or Beyond 0.28 0.28 0.01 1.14 0.27 0.25 0.02 4.96 
Taken Advanced Mathematics 0.49 0.49 0.01 1.59 0.50 0.49 0.01 1.80 
Taken Advanced Science 0.48 0.46 0.02 3.13 0.47 0.47 0.00 0.30 
Taken Advanced English 0.56 0.54 0.01 2.80 0.55 0.55 0.00 0.91 
First ACT Composite Score 22.63 22:75 -0.12 -2.44 22:53 22.46 0.07 1.44 
High School GPA 3.60 3:52 0.08 16.21 3.56 3.56 0.01 1.37 


Note: Standardized difference percentages are indexes that measure the effect size between two groups. Among propensity score literature a common criteria for concluding 
similarity of groups is a standardized difference percentage of less than 20%. For a more detailed treatment on standardized differences the reader is directed to Yang & Dalton 
(2012). 
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Figure 1. Distribution of Propensity Score for the Treatment and Control Groups. 
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Descriptive Statistics 


After identifying the matching control group via 
PSM, 2,660 students were used as the analytic 
sample to answer the first research question; the 
1,330 students who prepared for the second test 
were used to answer the second research 
question. These 1,330 students were the 
treatment condition for the first research 
question. Table 4 shows means and standard 
deviations for the variables of interest. Students 
who participated in test preparation before the 
retest had a slightly higher ACT Composite 
score relative to students who did not. 
Covariates are similar across the two groups. 


RQT1: The Impact of Test Preparation 
on Retest Scores 


Table 5 presents the full and reduced regression 
models used to determine if there was a 
statistically significant difference in the ACT 
Composite retest score between those students 
who prepared for the second test and those who 


did not. In the full model, results showed that 
there was no statistically significant interaction 
between first ACT score and whether the 
student prepared for the ACT. This means there 
were no differential effects of test preparation on 
students’ retest score by how well students did 
on the first test. There were also no significant 
interactions between preparation and 
race/ethnicity or gender. 


After controlling for student demographic 
information and academic performance, a 
statistically significant difference in the retest 
ACT Composite score for the two groups was 
found in both the full and reduced models. Using 
the reduced model, the adjusted ACT Composite 
means for the test preparation and non-test 
preparation groups were 24.33 and 23.63, 
respectively. Test preparation was the second 
strongest predictor of ACT Composite scores. 
The reduced model explained 87.5% of the 
variation in ACT Composite scores. 
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Table 4. Descriptive Statistics, by Test Preparation Status and Overall 


No Preparation Preparation Total Population 
(Cait 1!) (n=1,330) (n = 2,660) (N= 76,000) 


Test Preparation 0.00 1.00 0.00 0.50 
First ACT Composite Score 22.46 4.85 22.53 4.55 22.50 4.70 20.92 4.78 
Second ACT Composite Score® 23.33 4.98 24.12 4.83 23.72 4.92 22.02 4.99 
High School GPA 3.56 0.46 3.56 0.47 3.56 0.47 3.40 0.53 
Taken Algebra 2 or Beyond 0.58 0.49 0.56 0.50 0.57 0.50 0.50 0.50 
Taken Physics or Beyond 0.30 0.46 0.31 0.46 0.31 0.46 0.28 0.45 
Taken Advanced Mathematics 0.49 0.50 0.50 0.50 0.49 0.50 0.35 0.47 
Taken Advanced Science 0.47 0.50 0.47 0.50 0.47 0.50 0.34 0.47 
Taken Advanced English 0.55 0.50 0.55 0.50 0.55 0.50 0.41 0.49 
Skills Needed 0.53 0.50 0.54 0.50 0.54 0.50 0.62 0.48 
Public School 0.86 0.34 0.82 0.39 0.84 0.37 0.82 0.38 
Parents’ Income 
Low Income (<$36K) 0.24 - 0.22 - 0.23 - 0.30 - 
Middle Income ($36K-$100K) 0.47 - 0.48 - 0.47 - 0.43 - 
High Income (>$100kK) 0.29 - 0.30 - 0.30 - 0.27 - 
Parent’s Highest Education Level 
High School or Less 0.16 - 0:15 - 0:15 - 0.19 - 
Business/Technical School 0.04 - 0.04 - 0.04 - 0.04 - 
Some College, No Degree 0.10 - 0.11 - 0.11 - 0.12 - 
Associates Degree 0.10 - 0.10 - 0.10 - 0.10 - 
Bachelor’s Degree 0.31 - 0.31 - 0.31 - 0.29 - 
Graduate Study 0.21 - 0.21 - 0.21 - 0.17 - 
Doctorate or Professional 
Degree 0.08 - 0.08 - 0.08 - 0.08 - 
Race 
White 0.65 - 0.67 - 0.66 - 0.57 - 
Minority 0.22 - 0.21 - 0.21 - 0.30 - 
Asian 0.06 - 0.06 - 0.06 - 0.05 - 
Two or More Races 0.04 - 0.03 - 0.03 - 0.03 - 
No Response 0.04 - 0.03 - 0.03 - 0.04 - 
Gender (Male) 0.40 - 0.36 - 0.38 - 0.42 - 
Student Grade Level 
Grade 10 0.07 - 0.07 - 0.07 - 0.04 - 
Grade 11 0.84 - 0.86 - 0.85 - 0.81 - 
Grade 12 0.08 - 0.06 - 0.07 - 0.14 - 
Test Anxiety 0.24 0.43 0.24 0.43 0.24 0.43 - - 


Test Preparation Activities 
Practice Test in ACT’s Free Preparing 


for the ACT - - 0.74 0.96 0.37 0.77 - - 
ACT Online Prep - - 0.40 0.75 0.20 0.57 - - 
Another Web-Based Test-Preparation 
Program - - 0.39 0.80 0.19 0.60 - - 
The Real ACT Prep Guide - - 0.75 1.04 0.38 0.82 - - 
Another Test Prep Workbook - - 0.75 1.07 0.38 0.84 - - 
Test Prep Workshops/Courses Offered 
by the High School - - 0.69 1.10 0.35 0.85 - - 
Commercial Test Prep Course - - 0.31 0.80 0.15 0.58 - - 
Private Tutor or Consultant - - 0.41 0.89 0.20 0.66 - - 
One-on-One with a High School 
Teacher - - 0.30 0.67 Ot 0.50 - - 
Other Test Prep Software - - 0.23 0.62 0.11 0.46 - - 
Perception of Inadequate Preparation - - 0.28 0.45 0.32 0.47 - - 


Note: Test preparation activity items were only asked to students who indicated they prepared for the second test. 
“Approximately 12% of second ACT scores were missing for the population. 
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Table 5. Multiple Linear Regression Full and Reduced Models Predicting ACT Composite Scores from Test 


Preparation Status (n= 2,660). 


(Constant) 
Test Preparation 
First ACT Composite Score 


Test Preparation X First ACT Composite Score 


Taken Algebra 2 or Beyond 
Taken Physics or Beyond 
Taken Advanced Mathematics 
Taken Advanced Science 
Taken Advanced English 

High School GPA 


High School GPA X First ACT Composite Score 


Skills Needed 
Parents’ Income? 
Low Income (<$36K) 
High Income (>$100k) 
Fee Waiver 
Public School 
Parents’ Education” 
Business/Technical School 
Some College, No Degree 
Associates Degree 
Bachelor’s Degree 
Graduate Study 
Doctorate or Professional Degree 
Students’ Race‘ 
Minority 
Asian 
Two or More Races 
No Response 
Male 
Student Grade Level 
Test Anxiety 
Test Preparation X Asian 


Test Preparation X Two or More Races 


Test Preparation X No Response 
Test Preparation X Male 

Test Preparation X Low Income 
Test Preparation X High Income 


Note: Variables are grand mean centered. 


- dropped in the reduced model. 
*p <.05 


* Moderate income is the reference category 
> High School Diploma or less is the reference category 
°White is the reference category 


Full Model 
SE B 
0.04 

0.07 0.07 
0.01 0.82 
0.02 0.00 
0.08 0.02 
0.08 0.00 
0.10 0.02 
0.09 0.01 
0.09 0.01 
Gat: oor 
0.02 0.04 
0.08  -0.02 
0.11 0.00 
0.09 0.02 
0.11 -0.02 
0.10  -0.02 
0.19 0.01 
0.14 0.00 
0.15 0.01 
0.12 0.01 
0.14 0.01 
0.17 0.04 
0.10  -0.03 
0.15 -0.01 
0.19  -0.02 
0.20 0.00 
0.07 0.02 
0.10  -0.02 
0.08  -0.01 
0.30 0.00 
0.38 —--0.01 
0.39 0.02 
0.14 0.00 
0.19 0.00 
0.17 0.02 


t 
584.62 
10.30* 
76.35* 
0.27 
2.88* 
0.55 
2.022* 
A125. 
0.71 
6.96* 
4.632* 
-2.94* 


-0.22 
2.89" 

-1.85 

-3.02* 


1.20 
-0.31 
0.89 
0.78 
0.73 
3.914* 


-3.77 
-1.30* 
-2.68* 
0.41 
2.72* 
-3.18* 
-0.64 
-0.59 
-1.40 
2.780* 
0.41 
0.46 
2.625* 


nXzxe LU Lex=Xe "fete (=) 


SE 
0.04 
0.07 
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RQ2 and RQ3: Impact of Amount of 
Time in 10 Preparation Activities and 
I!l-Prepared Perceptions on Retest 
Scores 


To facilitate a better understanding of the 
context for the regression analysis that 
investigated which test preparation activities 
improved students’ ACT scores, we first provide 
descriptive information on the number of 
activities engaged in, by students’ demographic 
characteristics (Table 6). 


More than 65% of the students reported 
participating in one to three test preparation 
activities, and an additional 27% of students 
reported participating in four to seven activities. 
Very few students (7%) reported participating in 
eight or more activities. A trend emerged 
between key student demographic characteristics 
and the number of activities students participated 
in. Specifically, the percentage of minority 
students, percentage of fee waiver students, and 
percentage of students with low family income 
increased as the number of test preparation 
activities engaged in increased. 


A second linear regression analysis, including 
only those students who participated in test 
preparation, was conducted to determine 
whether the amount of time allocated to 10 test 
preparation activities had a positive impact on 
students’ retest score above the main effect of 
test preparation found in the first research 
question (Table 7). In addition, we looked to see 
if students’ perception of inadequate preparation 
impacts retest score given the importance of this 
variable as shown in the literature. 


Among the 10 test preparation activities 
investigated, only the number of hours working 
with a private tutor or consultant had a significant 
impact on retest scores above the overall effect 
for test preparation. Using the reduced model, 
those who did engage in this test preparation 
activity for 11 or more hours had an adjusted 
mean ACT Composite retest score that was 0.60 
points higher than students who did not use a 
private tutor or consultant. Pair-wise 
comparisons of the four bins that measured the 
amount of time allocated to a tutor or consultant 
showed that students who participated in 11 
hours or more with a tutor or consultant had a 
statistically higher ACT Composite re-test score 
than those who did not spend any hours on the 
activity. There were 280 students who 
participated in this test preparation activity, with 
106 students allocating 11 hours or more. No 
other comparisons were found statistically 
different. 


From the same model, perceived inadequate 
preparation was also a statistically significant 
predictor of students’ retest scores.* Students 
who self-reported feeling inadequately prepared 
for the second test had an ACT Composite score 
that was 0.322 scale score point below that of 
students who felt adequately prepared. The 
model explained 86.6% of the variation in ACT 
Composite scores.® 
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Table 6. Student Demographic Information, by Self-reported Number of Test Preparation Activities 


% % 
Number of Fee Average % 
activities Frequency AVEENIVe1 8 HSGPA White 
Unknown 10 60.00 30.00 3.61 60.00 
1-3 873 37.80 17.41 3.60 72.16 
4-7 359 31.48 22.01 3.50 57.94 
8+ 88 37.50 35.23 3.39 54.55 


% minority is African American, Hispanic, American Indian and Native Hawaiian. 


% 


Minority 


10.00 
16.72 
27.86 
32.95 


Race/Ethnicity 


i) 


% IXTo} 
Asian Response 
20.00 10.00 0.00 

5.96 3.55 1.60 

6.69 2:51 5.01 

4.55 3.41 4.55 


Family Income 


% % % 
(Lewy Middle High 
Tatere)ant=) Taterolaats Tatexo)nal=) 
50.00 40.00 10.00 
18.33 49.37 32.30 
27.30 46.52 26.18 
35.23 40.91 23.86 
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Table 7. Multiple Linear Regression Full and Reduced Model Predicting ACT Composite Scores from 
Amount of Time in 10 Test Preparation Activities (n= 1,330) 
Full Model NX-Yo LU Cex-Yo Mi KeXe[-1| 
b SE B t b SE B t 
(Constant) 23.98 0.06 412.72* 23.99 0.06 413.20 
First ACT Composite Score 0.88 0.02 0.83 54.40* 0.88 0.02 0.82 55.25* 
Taken Algebra 2 or Beyond 0.14 0.11 0.01 1.23" - - - - 
Taken Advanced Mathematics 0.37 0.12 0.04 3.20* 0.36 0.12 0.04 3:12* 
High School GPA 0.47 0.15 0.05 3.10* 0.48 O15 0.05 33177 
High School GPA X First ACT Composite Score 0.07 0.02 0.03 2.73* 0.07 0.03 0.03 2.69* 
Skills Needed -0.32 0.11 -0.03 -2.96* -0.32 0.114 -0.03 -2.92* 
Parents’ Income? 
Low Income (<$36K) -0.08 0.14 -0.01 -0.61 -0.08 0.14 -0.01 -0.58 
High Income (>$100k) 0.41 0.12 0.04 3.49* 0.44 O12 0.04 3.68* 
Public School -0.34 0.13  -0.03 -2.59* -0.37 0.13 -0.03 -2.85* 
Students’ Race? 
Minority -0.58 0.14 -0.05 -4.21* -0.57 0.14 -0.05 -4.13* 
Asian -0.25 0.21 = -0.01 -1.18 -0.23 0.21 -0.01 -1.09 
Two or More Races -0.80 0.28 -0.03 -2.88* -0.78 0.28 -0.03 -2.82* 
Race Category No Response 0.46 0.31 0.02 1.50 0.59 0.31 0.02 1.92 
Male 0.20 0.10 0.02 1.97* 0.19 0.10 0.02 1.80 
Student Grade Level -0.37. 0.14 -0.03 -2.63* -0.37 0.14 -0.03 -2.62* 
Test Preparation Activities (amount of time) 
Practice Test in ACT’s Free Preparing for the 
ACT 0.01 0.06 0.00 0.21 - - - - 
ACT Online Prep -0.02 0.07 0.00 -0.27 - - - - 
Another Web-Based Test-Preparation Program 0.04 0.07 0.01 0.49 - - - - 
The Real ACT Prep Guide 0.10 0.05 0.02 1.98 - - - - 
Another Test Prep Workbook 0.10 0.05 0.02 1.95 - - - - 
Test Prep Workshops/Courses Offered by the 
High School 0.08 0.05 0.02 1.76 - - - - 
Commercial Test Prep Course 0.01 0.07 0.00 0.14 - - - - 
Private Tutor or Consultant 0.16 0.06 0.03 2:70" 0.20 0.06 0.04 3.53* 
One-on-One with a High School Teacher -0.12 0.09 -0.02 -1.35 - - - - 
Other Test Prep Software -0.04 0.09 -0.01 -0.46 - - - - 
Perception of Inadequate Preparation -0.28 0.11 -0.03 -2.48* -.322 113 -.030 -2.847* 


*p <.05 


Note: Variables are grand mean centered. The reduced model for RQ1 was used as a starting point for answering RQ2. 


- dropped in the reduced model. 
* Moderate income is the reference category 
> White is the reference category 


Discussion 


We examined the impact of using test 
preparation prior to taking the ACT a second 
time. Emphasis was placed on those students 
who did not use test preparation materials 
before taking the ACT the first time. This line of 
research is critical given the potential 


inequalities in using test preparation products for 


college entrance examinations. There is no 


question that the use of standardized tests to aid 


college admissions decisions creates a very 


high stakes situation for graduating high school 
students. It is because of this fact that the test 
preparation industry has seen such strong 
growth in the United States (Barnes Reports, 
2017). It is also the reason why a NACAC 
discussion paper (Briggs, 2009) has called for 
more rigorous research to be conducted on the 
effects of test preparation for the ACT. The 
present research study contributes to this 
needed body of evidence. 


This study uncovered a statistically significant 
impact of test preparation on ACT retest scores. 
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On average, those who participated in test 
preparation prior to taking the ACT a second 
time had an ACT Composite score 0.71 scale 
score points above those who did not. This is 
consistent with prior test preparation studies that 
have found a positive, yet small, effect of test 
preparation on score gains. 


Additionally, we found that the effects of test 
preparation did not differ by how well students had 
done on a prior administration of the test. Most 
importantly, we also found that the effects of test 
preparation did not differ by race/ethnicity, gender, 
or family income. This suggests that test 
preparation is equally effective for minority 
students as for majority students and equally 
effective for females as for males. The findings 
pertaining to family income suggest that financial 
means, and any associated access to test 
preparation activities, overall, did not impact ACT 
retest scores.’ 


When investigating whether the amount of time 
allocated to specific types of test preparation 
activities improve ACT retest scores above the 
main effect found for test preparation overall, we 
found that working with a private tutor or 
consultant improved students' ACT retest scores. 
This was true specifically for students who did so 
for 11 hours or more, which included a relatively 
small number of students. Regardless, this type of 
one-on-one, personalized activity was found to 
have a notable effect on retest scores. This could 
have important practical significance in the 
admissions process for schools who utilize cut 
scores when evaluating prospective students 
(Briggs, 2009). 


At first blush, this finding may seem to contradict 
the finding that family income did not have a 
differential effect on test preparation efficacy, as 
families with higher incomes may be more likely to 
be able to afford private tutoring.8 The two findings 
can be reconciled by a nuanced understanding of 
what is being tested by the different models and 
samples employed in the current study. In the first 
model, we examined the effect of family income 
on test preparation efficacy regardless of the type 
of test preparation used or the duration of time 
spent on those activities for students who used 
test preparation versus students who did not use 


any test preparation. In the second model, we 
examined the impact of the number of hours 
using specific test preparation activities while 
controlling for the effects of family income only on 
students who engaged in at least one test 
preparation activity. This finding therefore 
suggests that, regardless of family income, 

private tutoring has a positive effect on ACT retest 
scores. 


In future research, we believe it will be beneficial 
to further tease out the impacts of participation in 
specific test preparation from the intensity of 
participating in that activity while considering 
access issues (e.g. financial status). Due to the 
relatively small number of low- and middle-income 
students who reported working with private tutors, 
we were unable to fully examine the interaction 
between family income and the number of hours 
working with a private tutor or consultant. We 
would be remiss if we did not emphasize that this 
line of inquiry is needed in future work to further 
understand how access to and use of a private 
tutor might lead to inequality issues for low- 
income students. Further, we recommend that 
future research use a continuous measure for the 
amount of time allocated to different test 
preparation activities and consider using 
measures of test preparation other than self- 
reported indicators. 


A final notable result of this study was that 
students’ self-perception of preparation for the 
ACT test was a significant predictor of retest 
performance. This study highlights students’ own 
understanding of their academic standings in 
relation to content mastery and potential 
performance on tests. It would be beneficial to 
utilize this type of self-knowledge to help connect 
students with study aids that can help them raise 
their perceived preparation by increasing their 
actual preparation for the ACT test. 


While we focused on test preparation activities 
that occurred after the first test but before the 
second and investigated the number of hours of 
test preparation engagement, we did not measure 
the time elapsed between engaging in the test 
preparation activity and completing the ACT. 
Future research might investigate this relationship 
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and address questions like: How should the time supplementary instruction to strengthen skills 

allocated to test preparation activities be and higher-achieving students who are using 

distributed across the year? Is there an optimal test preparation materials to review content and 

time in which test preparation should begin? increase their chances of getting an exceptional 
score. If we have more high-achieving students 

This study took an important step forward in in our study sample, we need to be cognizant of 

providing causal evidence for the impact test the potential impacts to generalizability. 

preparation has on standardized tests such as 

the ACT. We have to make note of the fact that In sum, this study contributes to the test 

study participants had an average ACT preparation literature by evaluating the effects of 

Composite test score that was 1.6 scale score test preparation on the ACT which, as Briggs 

points above the national average. This is worth (2009) has noted, is sorely needed. Moreover, 

considering because when we speak of students unlike the typical study of test preparation, this 

who use test preparation materials, we can study employed a rigorous research 

imagine different types of test preparation users, methodology that allowed for a causal 

including lower-achieving students who need examination of efficacy. 

Notes 


1. 


2. 
3. 


For example, this study found that roughly 13 hours of individual tutoring is associated with an increase of 30 
points or an SAT Total score gain of 0.25 SD. 

Listwise deletion was used to handle missing data. 

To check for the effect of shared variance due to the matching performed, cluster robust standard errors that 
accounted for the paired clusters were implemented in a separate model. Accounting for this shared variance 
did not dramatically impact model estimates. For example, the maximum difference in estimates for the reduced 
model in research question 1 was 0.01223. The proportion of variance explained is predominantly driven by prior 
test score. The proportion of variance explained without test prep was 0.866. With test prep included in the 
model the proportion of variance explained increased to 0.871. 

Among students who reported using one of the 10 activities in question, the correlation between number of 
activities undertaken and self-reported feeling of being underprepared to take the ACT was -0.032 (p=0.2365). 
Due to the low correlation, it is unlikely that the number of test preparation activities engaged in mediates the 
effect of under-preparation on ACT Composite score. 

The proportion of variance explained is predominantly driven by prior test score. The proportion of variance 
explained without test prep was 0.864. With test prep included in the model, the proportion of variance explained 
increased to 0.865.0. 

While there was a significant interaction between test preparation and high income in the full model, this was not 
the case in the reduced model. 

In the current sample, the Spearman correlation between family income and number of hours spent working with 
a private tutor or consultant was relatively small 0.078 (p<0.0001), suggesting that there is a weak relationship 
between income and hours using a private tutor or consultant. 
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